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WHAT IS CLAIMED IS: 

1 . A system for generating a description record from image 
information, comprising: 

at least one image input interface for receiving said image information; 

a computer processor coupled to said at least one image input interface 
for receiving said image information therefrom, processing said image information by 
performing image object extraction processing to generate image object descriptions 
from said image information, processing said generated image object descriptions by 
object hierarchy construction and extraction processing to generate image object 
hierarchy descriptions, and processing said generated image object descriptions by 
entity relation graph generation processing to generate entity relation graph 
descriptions, wherein at least one description record including said image object 
descriptions, said image object hierarchy descriptions and said entity relation graph 
descriptions is generated to represent content embedded within said image 
information; and 

a data storage system, operatively coupled to said processor, for storing 
said at least one description record. 

2. The system of claim 1 , wherein said image object extraction 
processing and said object hierarchy construction and extraction processing are 
performed in parallel. 

3. The system of claim 1 , wherein said image object extraction 
processing comprises: 

image segmentation processing to segment each image in said image 
information into regions within said image; and 

feature extraction and annotation processing to generate one or more 
feature descriptions for one or more said regions; 
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whereby said generated image object descriptions comprise said one or 
more feature descriptions for one or more said regions. 

4. The system of claim 3, wherein said one or more feature 
descriptions are selected from the group consisting of media features, visual features, 
and semantic features. 

5. The system of claim 4, wherein said semantic features are 
further defined by at least one feature description selected from the group consisting 
of who, what object, what action, where, when, why, code downloading, and text 
annotation. 

6. The system of claim 4, wherein said visual features are further 
defined by at least one feature description selected from the group consisting of color, 
texture, position, size, shape, motion, code downloading, and orientation. 

7. The system of claim 4, wherein said media features are further 
defined by at least one feature description selected from the group consisting of file 
format, file size, color representation, resolution, data file location, author, creation, 
scalable layer, code downloading, and modality transcoding. 

8. The system of claim 1, wherein said object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on visual feature relationships of image 
objects represented by said image object descriptions. 

9. The system of claim 1, wherein said object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on semantic feature relationships of image 
objects represented by said image object descriptions. 
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10. The system of claim 1 , wherein said object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on media feature relationships of image 
objects represented by said image object descriptions. 

1 1 . The system of claim 1 , wherein said object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on relationships of image objects represented 
by said image object descriptions, wherein said relationships are selected from the 
group consisting of visual feature relationships, semantic feature relationships and 
media feature relationships. 

12. The system of claim 1 , wherein said object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on relationships of image objects represented 
by said image object descriptions, wherein said image object hierarchy descriptions 
have a plurality of hierarchical levels. 

13. The system of claim 12, wherein said image object hierarchy 
descriptions having a plurality of hierarchical levels comprise clustering hierarchies. 

14. The system of claim 13, wherein said clustering hierarchies are 
based on relationships of image objects represented by said image object descriptions, 
wherein said relationships are selected from a group consisting of visual feature 
relationships, semantic feature relationships and media feature relationships. 

15. The system of claim 1, wherein said entity relation graph 
generation processing generates entity relation graph descriptions of said image object 
descriptions based on relationships of image objects represented by said image object 
descriptions, wherein said relationships are selected from the group consisting of 
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visual feature relationships, semantic feature relationships and media feature 
relationships. 

16. The system of claim 1, further comprising an encoder for 
receiving and encoding said image object descriptions into encoded description 

5 information, wherein said data storage system is operative to store said encoded 
description information as said at least one description record. 

17. The system of claim 1, wherein said image object descriptions, 
said image object hierarchy descriptions, and said entity relation graph descriptions 
are combined together to form image descriptions, and further comprising an encoder 

1 0 for receiving and encoding said image descriptions into encoded description 

information, wherein said data storage system is operative to store said encoded 
description information as said at least one description record. 

1 8. The system of claim 17, wherein said encoder comprises a 
binary encoder. 

15 19. The system of claim 1 7, wherein said encoder comprises an 

XML encoder. 

20. A method for generating a description record from image 
information, comprising the steps of: 

receiving said image information; 
20 processing said image information by performing image object 

extraction processing to generate image object descriptions from said image 
information; 

processing said generated image object descriptions by object 
hierarchy construction and extraction processing to generate image object hierarchy 
25 descriptions; 
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processing said generated image object descriptions by entity relation 
graph generation processing to generate entity relation graph descriptions, wherein at 
least one description record including said image object descriptions, said image 
object hierarchy descriptions and said entity relation graph descriptions is generated to 
5 represent content embedded within said image information; and 

storing said at least one description record. 

2 1 . The method of claim 20, wherein said steps of image object 
extraction processing and object hierarchy construction and extraction processing are 
performed in parallel. 

22. The method of claim 20, wherein said step of image object 
extraction processing comprises the further steps of: 

image segmentation processing to segment each image in said image 
information into regions within said image; and 

feature extraction and annotation processing to generate one or more 
feature descriptions for one or more said regions; 

whereby said generated image object descriptions comprise said one or 
more feature descriptions for one or more said regions. 

23. The method of claim 22, further comprising the step of 
selecting said one or more feature descriptions from the group consisting of media 

20 features, visual features, and semantic features. 

24. The method of claim 23, wherein said semantic features are 
further defined by at least one feature description selected from the group consisting 
of who, what object, what action, where, when, why, code downloading, and text 
annotation. 



10 



15 
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25. The method of claim 23, wherein said visual features are further 
defined by at least one feature description selected from the group consisting of color, 
texture, position, size, shape, motion, code downloading, and orientation. 

26. The method of claim 23, wherein said media features are 
further defined by at least one feature description selected from the group consisting 
of file format, file size, color representation, resolution, data file location, author, 
creation, scalable layer, code downloading, and modality transcoding. 

27. The method of claim 20, wherein said step of object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on visual feature relationships of image 
objects represented by said image object descriptions. 

28. The method of claim 20, wherein said step of object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on semantic feature relationships of image 
objects represented by said image object descriptions. 

29. The method of claim 20, wherein said step of object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on media feature relationships of image 
objects represented by said image object descriptions. 

30. The method of claim 20, wherein said step of object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on relationships of image objects represented 
by said image object descriptions, wherein said relationships are selected from the 
group consisting of visual feature relationships, semantic feature relationships and 
media feature relationships. 
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3 1 . The method of claim 20, wherein said step of object hierarchy 
construction and extraction processing generates image object hierarchy descriptions 
of said image object descriptions based on relationships of image objects represented 
by said image object descriptions, wherein said image object hierarchy descriptions 

5 are configured to have a plurality of hierarchical levels. 

32. The method of claim 31, wherein said image object hierarchy 
descriptions having a plurality of hierarchical levels are configured to comprise 
clustering hierarchies. 

33. The method of claim 32, wherein said clustering hierarchies are 
10 configured to be based on relationships of image objects represented by said image 

object descriptions, wherein said relationships are selected from a group consisting of 
visual feature relationships, semantic feature relationships and media feature 
relationships. 

34. The method of claim 20, wherein said step of entity relation 

1 5 graph generation processing generates entity relation graph descriptions of said image 
object descriptions based on relationships of image objects represented by said image 
object descriptions, wherein said relationships are selected from the group consisting 
of visual feature relationships, semantic feature relationships and media feature 
relationships. 

20 35. The method of claim 20, further comprising the steps of 

receiving and encoding said image object descriptions into encoded description 
information, and storing said encoded description information as said at least one 
description record. 

36. The method of claim 20, further comprising the steps of 
25 combining said image object descriptions, said image object hierarchy descriptions, 
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and said entity relation graph descriptions to form image descriptions, and receiving 
and encoding said image descriptions into encoded description information, and 
storing said encoded description information as said at least one description record. 



37. The method of claim 36, wherein said step of encoding 
5 comprises the step of binary encoding. 

38. The method of claim 36, wherein said step of encoding 
comprises the step of XML encoding. 

39. A computer readable media containing digital information with 
at least one description record representing image content embedded within 

10 corresponding image information, the at least one description record comprising: 
one or more image object descriptions generated from said image 
information using image object extraction processing; 

one or more image object hierarchy descriptions generated from said 
generated image object descriptions using object hierarchy construction and extraction 
15 processing; and 

one or more entity relation graph descriptions generated from said 
generated image object descriptions using entity relation graph generation processing. 



40. The computer readable media of claim 39, wherein said image 
object descriptions, said image object hierarchy descriptions, and said entity relation 
20 graph descriptions further comprise one or more feature descriptions. 



41 . The computer readable media of claim 40, wherein said one or 
more feature descriptions are selected from the group consisting of media features, 
visual features, and semantic features. 
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The computer readable media of claim 41 , wherein said 



semantic features are further defined by at least one feature description selected from 
the group consisting of who, what object, what action, where, when, why, code 
downloading, and text annotation. 



features are further defined by at least one feature description selected from the group 
consisting of color, texture, position, size, shape, motion, code downloading, and 
orientation. 

44. The computer readable media of claim 4 1 , wherein said media 
10 features are further defined by at least one feature description selected from the group 

consisting of file format, file size, color representation, resolution, data file location, 
author, creation, scalable layer, code downloading, and modality transcoding. 

45. The computer readable media of claim 39, wherein said object 
hierarchy descriptions are based on visual feature relationships of image objects 

1 5 represented by said image object descriptions. 

46. The computer readable media of claim 39, wherein said image 
object hierarchy descriptions are based on semantic feature relationships of image 
objects represented by said image object descriptions. 

47. The computer readable media of claim 39, wherein said image 
20 object hierarchy descriptions are based on media feature relationships of image 

objects represented by said image object descriptions. 



object hierarchy descriptions are based on relationships of image objects represented 



43. 



The computer readable media of claim 41, wherein said visual 



48. 



The computer readable media of claim 39, wherein said image 
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by said image object descriptions, wherein said image object hierarchy descriptions 
have a plurality of hierarchical levels. 

49. The computer readable media of claim 48, wherein said image 
object hierarchy descriptions having a plurality of hierarchical levels comprise 

5 clustering hierarchies. 

50. The computer readable media of claim 49, wherein said 
clustering hierarchies are based on relationships of image objects represented by said 
image object descriptions, wherein said relationships are selected from a group 
consisting of visual feature relationships, semantic feature relationships and media 

1 0 feature relationships. 

5 1 . The computer readable media of claim 39, wherein said entity 
relation graph descriptions are based on relationships of image objects represented by 
said image object descriptions, wherein said relationships are selected from the group 
consisting of visual feature relationships, semantic feature relationships and media 

1 5 feature relationships. 

52. The computer readable media of claim 39, wherein said image 
object descriptions are in the form of encoded description information. 

53. The computer readable media of claim 39, wherein said image 
object descriptions, said image object hierarchy descriptions, and said entity relation 

20 graph descriptions are combined together in the form of encoded description 
information. 

54. The computer readable media of claim 53, wherein said 
encoded description information is in the form of binary encoded information. 
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55. The computer readable media of claim 53, wherein said 
encoded description information is in the form of XML encoded information. 

56. The system of claim 12, wherein said image object hierarchy 
descriptions having a plurality of hierarchical levels are configured to comprise 

5 multiple levels of abstraction hierarchies. 

57. The system of claim 56, wherein said multiple levels of 
abstraction hierarchies are configured to be based on relationships of image objects 
represented by said image object descriptions, wherein said relationships are selected 
from a group consisting of visual feature relationships, semantic feature relationships, 

1 0 and media feature relationships. 

58. The method of claim 3 1 , wherein said image object hierarchy 
descriptions having a plurality of hierarchical levels are configured to comprise 
multiple levels of abstraction hierarchies. 

59. The method of claim 58, wherein said multiple levels of 

15 abstraction hierarchies are configured to be based on relationships of image objects 
represented by said image object descriptions, wherein said relationships are selected 
from a group consisting of visual feature relationships, semantic feature relationships, 
and media feature relationships. 

60. The computer readable media of claim 48, wherein said image 
20 object hierarchy descriptions having a plurality of hierarchical levels are configured to 

comprise multiple levels of abstraction hierarchies. 

61 . The computer readable media of claim 60, wherein said 
multiple levels of abstraction hierarchies are configured to be based on relationships 
of image objects represented by said image object descriptions, wherein said 
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semantic feature relationships, and media feature relationships. 



62. 



The system of either claim 3 or 4, wherein said one or more 



feature descriptions include pointers to extraction and matching code in order to 
5 facilitate code downloading. 

63. The method of either claim 22 or 23, wherein said one or more 
feature descriptions include pointers to extraction and matching code in order to 
facilitate code downloading. 

64. The computer readable media of either claim 40 or 41, wherein 
1 0 said one or more feature descriptions include pointers to extraction and matching code 

in order to facilitate code downloading. 
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